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DETAILED ACTION 
Remarks 

1. In response to communications filed on 29-April-2004, new claims 21-23 have been added 
per applicant's request. 

2. In view of the amendment faxed on 27-August-2004, authorized to be entered as an 
"examiner's amendment", claims 4-5 are canceled, and claims 1, 11, 14, and 19 are amended 
per applicant's request. Therefore, claims 1-3 and 7-23 are presently pending in the 
application, of which, claims 1,7, 11, 14, and 19 are in independent format. 

Examiner's Amendment 

3. An examiner's amendment to the record appears below. Should the changes and/or additions 
be unacceptable to appUcant, an amendment may, be filed as provided by 37 CFR 1.312. To 
ensure consideration of such an amendment, it MUST be submitted no later than the payment 
of the issue fee. 

Authorization for this examiner's amendment was given in a telephone interview with 
Mr. Samuel Ntiros (Attorney of Record) on 27-August-2004, and via the "amendment 
authorized to be entered by examiner", faxed on 27-August-2004 (see enclosed Interview 
Summary, dated 27-August-2004, for details.) 
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Claims 1, 4-5, 1 1, 14, and 19 have been amended by the examiner as follows. A new 
listing of the claims is provided in this Office Action (in paragraph 6), following the Reasons 
for Allowance: 

Claim 1 (Currently Amended): 

A multimedia data structure embodied in a computer readable medium and used for an image 
search, comprising: 

(a) mformation describmg at least one feature of a certain image; 

(b) recent user feedback information based on user relevance feedback; [and] 

(c) whole feedback information based on the user relevance feedback obtained since 
formation of the data structure; 

recent user feedback reliabilitv information representing how reliable the recent user 
feedback information is: and 

whole feedback reliabilitv information representing how reliable the whole feedback 
information is, wherein the recent user feedback reliabilitv information is expressed bv 



where. N is a number of user feedback responses, m is a number of images in a similar image 
list, and a is a number of user feedback responses given to the i-th image. 




N 
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Claim 4 (Currently Amended): 

Canceled. 

Claim 5 (Currently Amended): 

Canceled. 

Claim 11 (Currently Amended): 

A [The] multimedia data structure [as claimed in claim 1] embodied in a computer 
readable medium and used for an image search, comprising: 
information describing at least one feature of a certain image: 
recent user feedback information based on user relevance feedback: 
whole feedback information based on the user relevance feedback obtained since 
formation of the data structure, wherein the recent user feedback information is represented 
as a similar image list, and the similar image list has an image list structure which includes a 
similar image identification, a score reflecting the current feedback, and a waiting duration 
representing a time period between a final feedback time point and a present time point. 
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Claim 14 (Currently Amended): 

A method of determining weights of image features used for an image search based on 
user relevance feedback, comprising: 

(a) providing a multimedia data structure including information describing the features 
of a certain image, recent user feedback information for the image, and whole feedback 
information for the image based on user relevance feedback obtained since formation of the 
data structure, and reliability information corresponding to the recent user feedback 
information and whole feedback information; 

(b) updating the recent user feedback information and whole feedback information and 
their reUabilities by learning them in response to the user relevance feedback; 

(c) determining weights of the image features in proportion to the reliabilities of the 
recent feedback information, the whole feedback information, or both the recent feedback 
information and the whole feedback information : wherein .the reliability of the recent user 
feedback information is expressed by 



where. N is a number of user feedback responses, m is a number of images i n a similar image 
list, and n; is a number of user feedback responses given to the i-th image. 
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Claim 19 (Currently Amended): 

A multimedia data structure embodied in a computer readable medium and reflecting 
change of a user relevance feedback for determining weights of image features used for an 
image search, comprising: 

(a) information describing at least one feature of a certain image; 

(b) recent user feedback information based on user relevance feedback; 

(c) whole feedback information based on the user relevance feedback obtained since 
formation of the data structure; and 

(d) reUability information indicating reUability of at least one of the user feedback 
information and whole feedback informatio n, wherein the recent user feedback information is 
represented as a similar image hst, and the similar image list has an image list structure 
which includes a similar image identification, a score reflecting the current feedback, and a 
waiting duration representing a time period between a fiflal feedback time point and a present 
time point. 

Allowance 

4. Claims 1-3 and 7-23 are allowed over the prior art made of record. 

5. The following is an examiner's statement of reasons for allowance: 

The appUcant's amendment, authorized by the attorney to be entered as an "examiner's 
amendment", filed on 27-August-2004, overcomes the cited prior art with respect to the 
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independent claims. Independent claims 1, 7, 1 1, 14, and 19 have been amended to 
incorporate "allowable subject matter", indicated in the previous office action. 

The prior art of record, Ma et al (U.S. Patent No. 6,347,3 13), Lidd y etal (U.S. Patent No. 
6,304,864), Herz (U.S. Patent No. 6,460,036), and Cohen (U.S. Patent No. 6,067,539) do not 
disclose, teach, or suggest the claimed Umitations of (in combination with all other features 
in the claims): 

wherem the recent user feedback reUability information is expressed by 



where, N is a number of user feedback responses, m is a number of images in a similar image 
list, and ni is a number of user feedback responses given to the i-th image, as claimed in 
claims 1 and 14. 

Claims 2-3 and 21-23 are allowed over prior art made of record because they are 
dependents from the allowed independent claim 1. 




Claims 15-18 are allowed over the prior art made of record because they are dependents 
from the allowed independent claims 14. 



Application/Control Number: 09/666,28 1 Page 8 

Art Unit: 2175 

The prior art of record, Ma et al (U.S. Patent No. 6,347,3 13), Liddv et al (U.S. Patent No. 
6,304,864), Herz (U.S. Patent No. 6,460,036), and Cohen (U.S. Patent No. 6,067,539) do not 
disclose, teach, or suggest the claimed limitations of (in combination with all other features 
in the claims): 

(b) if it is checked that the similar image does not exist in the current queue, inputting the 
similar image to an uppermost space of a queue entrance, setting a number of feedback 
responses of the similar image to "1", and shifting images existing in the queue to lower 
positions by one space; 

(c) if it is checked that the similar image exists in the current queue, increasing the 
number of feedback responses of the similar image, and shifting other images in the queue to 
upper positions by "N"; and 

(d) if any image is shifted to a lower position over a size of the queue at the respective 
steps, deleting the image shifted to the lower position from the queue, as claimed in claim 7. 

- - Claims 8-10 are allowed over the prior art made of record because they arej^ependent^ ^^^^^ 
from the allowed independent claim 7. 



The prior art of record, Ma et al (U.S. Patent No. 6,347,3 13), Liddv et al (U.S. Patent No. 
6^304,864), Herz (U.S. Patent No. 6,460,036), and Cohen (U.S. Patent No, 6,067,539) do not 
disclose, teach, or suggest the claimed Umitations of (in combination with all other features 
in the claims): 
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wherein the recent user feedback information is represented as a similar image list, and 
the similar image list has an image list structure which includes a similar image 
identification, a score reflecting the current feedback, and a waiting duration representing a 
time period between a final feedback time point and a present time point, as claimed in 
claims 11 and 19, 

Claims 12-13 are allowed over the prior art made of record because they are dependents 
fi-om the allowed independent claim 11. 

Claim 20 is allowed over the prior art made of record because it is dependent from the 
allowed independent claim 19. 

Listing of the Claims 

6. The following is a listing of the claims, as amended hy the examiner. This listing of claims 
will replace all prior versions, and Ustings of claims in the Application: 

1 , (Currently Amended) A multimedia data structure embodied in a computer readable 
medium and used for an image search, comprising: 

(a) information describing at least one feature of a certain image; 

(b) recent user feedback information based on user relevance feedback; [and] 
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(c) whole feedback information based on the user relevance feedback obtained since 
formation of the data structure; 

recent user feedback reHabiiitv information representing how reliable the recent user 
feedback information is: and 

whole feedback reliability information representing how reliable the whole feedback 
information is. wherein the recent user feedback reliability information is expressed by 



N 

where, N is a number of user feedback responses, m is a number of images in a similar image 
list, and n; is a number of user feedback responses given to the i-th image. 

2. (Previously presented) The multimedia data structure as claimed in claim 1, wherein 

the recent user feedback information is determined for a predetermined time period or by a 

predetermined frequency. 

3. (Previously presented) The multimedia data structure as claimed in claim 1, wherein 
the recent user feedback information includes a weight value learned by the user relevance 
feedback or similar image information, and the whole feedback information is represented by 
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a weight value learned by the user relevance feedback given since formation of the data 
structure. 

4. (Currently Amended) Canceled. 

5. (Currently Amended) Canceled. 

6. (Previously Canceled.) 

7. (Previously Presented) A method of managing image information for an image search, 
comprising: 

forming a data structure which includes information describing features of a certain 
image, recent user feedback information, and whole feedback information; and 

determining weights for the features of the certain image based on the recent user 
feedback information and the whole feedback information, , . 

wherein the method further comprises: 

representing the recent user feedback information by a similar image list, and 
reflecting a recent user feedback pattern based by the similar image list using a queue 
algorithm, and 

wherein the queue algorithm comprises: 

(a) checking whether a similar image exists in a current queue when the similar image is 
fed back; 
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(b) if it is checked that the similar image does not exist in the current queue, inputting the 
similar image to an uppermost space of a queue entrance, setting a number of feedback 
responses of the similar image to "1", and shifting images existing in the queue to lower 
positions by one space; 

(c) if it is checked that the similar image exists in the current queue, increasing the 
number of feedback responses of the similar image, and shifting other images in the queue to 
upper positions by "N"; and 

(d) if any image is shifted to a lower position over a size of the queue at the respective 
steps, deleting the image shifted to the lower position from the queue. 

8. (Previously presented) The method as claimed in claim 7, wherein "N'' is fixed or 
variably determined in accordance with the similar image, 

9. (Previously presented) The method as claimed in claim 7, wherein "N" is varied in 
inverse proportion to a frequency of appearance of the similar image. 

10. (Previously presented) The method as claimed in claim 7, wherein if the similar 
image does not exist in the queue when the similar image is fed back, the similar image is 
inputted to the queue only in case that the similar image is fed back more than a specified 
threshold number. 
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11. (Currently Amended) A [The] multimedia data structure [as claimed in claim 1] 
embodied in a computer readable medium and used for an image search, comprising: 

information describing at least one feature of a certain image: 
recent user feedback information based on user relevance feedback: 
whole feedback information based on the user relevance feedback obtained since 
formation of the data structure, wherein the recent user feedback information is represented 
as a similar image list, and the similar image list has an image list structure which includes a 
similar image identification, a score reflecting the current feedback, and a waiting duration 
representing a time period between a final feedback time point and a present time point. 

12. (Previously presented) A similar image list managing method for the multimedia data 
structure recited in claim 1 1, comprising: 

managing a similar image list in a manner that only fmages whose scores are over a 
predetermined threshold number are maintained in the list, or only images whose scores are 
on N upper positions of the list if a size of the hst is N. ^ 

13. (Previously presented) A feedback reflecting score updating method for the 
multimedia structure recited in claim 11, comprising updating image scores in a manner that 

(a) with respect to image fed back from similar images, a following calculation is 
performed: Score (new) = Score (current) x fW (Waiting Duration) + 1; and 

(b) with respect to other images not fed back, a following calculation is performed: Score 
(new) = Score (current) x fW (Waiting Duration); and wherein fW (Waiting Duration) is a 
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function having a characteristic that it returns smaller values as the waiting duration becomes 
longer. 

14. (Currently Amended) A method of determining weights of image features used for 
an image search based on user relevance feedback, comprising: 

(a) providing a multimedia data structure including information describing the features 
of a certain image, recent user feedback information for the image, and whole feedback 
information for the image based on user relevance feedback obtained since formation of the 
data structure, and reliability information corresponding to the recent user feedback 
information and whole feedback information; 

(b) updating the recent user feedback information and whole feedback information and 
their reliabilities by learning them in response to the user relevance feedback; 

(c) determining weights of the image features in proportion to the reliabilities of the 
recent feedback information, the whole feedback information, or both the recent feedback 
information and the whole feedback information : wherein the reliability of the recent user 
feedback information is expressed by 



where. N is a number of user feedback responses, m is a number of images in a similar image 
list, and nj is a number of user feedback responses given to the i-th image. 



1- - 




N 
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15. (Previously presented) The method as claimed in claim 14, wherein the recent user 
feedback information is represented by a weight value learned by the user relevance feedback 
or a similar image information, and the whole feedback information is represented by a 
weight value learned by feedback given since formation of the data structure. 

16. (Previously presented) The method as claimed in claim 14, wherein the reUability of 
the recent user feedback information is determined in proportion to a consistency of a 
recently used pattern or feedback. 

17. (Previously presented) The method as claimed in claim 14, wherein the reUability of 
the whole feedback information is determined in proportion to a number of feedback 
responses concerned in learning, 

18. (Previously presented) The method as claimed in claim 14, wherein the reliability of 
-the whole feedback information is responsive to recorded user usage records wherein the „ 

recorded user usage records provide feedback to the reliability of the whole feedback 
information without user interaction. 

19. (Currently Amended) A multimedia data structure embodied in a computer 
readable medium and reflecting change of a user relevance feedback for determining weights 
of image features used for an image search, comprising: 

(a) information describing at least one feature of a certain image; 
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(b) recent user feedback information based on user relevance feedback; 

(c) whole feedback information based on the user relevance feedback obtained since 
formation of the data structure; and 

(d) reliability information indicating reliability of at least one of the user feedback 
information and whole feedback information , wherein the recent user feedback information-is 
represented as a similar image list, and the similar image list has an image list structure 
which includes a si milar imag e identification, a score reflecting the current feedback, and a 
waiting duration representing a time period between a final feedback time p oint and a present 
time point. 

20. (Previously presented) The multimedia data structure of claim 19, wherein the 
reUability information indicates reliability of both of the user feedback information and the 
whole feedback information, 

21. (Previously presented) The multimedia data structure as claimed in claim 1, wherein 
the recent user feedback information and whole feedback information provide an indication 
of relevance of the information describing the at least one feature of the certain image. 

22. (Previously presented) The multimedia data structure of claim 1, wherein the at least 
one feature includes at least one of a color histogram, a texture histogram, and a dominant 
color of the image. 
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23. (Previously presented) The multimedia data structure of claim 1, wherein the 
information provides a statistical description of the image. 



7. Any inquiries concerning this communication or earlier communications from the examiner 
should be directed to Tony Mahmoudi whose telephone number is (703) 305-4887. The 
examiner can normally be reached on Mondays-Fridays from 08:00 am to 04:30 pm. 

If attempts to reach the examiner by telephone are unsuccessftil, the examiner's 
supervisor, Dov Popovici, can.be reached at (703) 305-3830. 



Conclusion 



tm 




August 30, 2004 



CHARLES RONES 
ooiMARY EXAMIISIER 



